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Abstract 

Given a Probabilistic Finite Automata (PFA), a set of states S, and an error threshold e > 0, our algorithm approxi- 
mates the infimum probability (quantifying over all infinite words) that the automata reaches S. Our result contrasts 
with the known result that the approximation problem is undecidable if we consider the supremum instead of the in- 
fimum. Since we study the probability of reaching a set of states, instead of the probability of ending in an accepting 
state, our work is more related to model checking than to formal languages. 
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1. Introduction 

Suppose you want to analyse a system J{ whose num- 
ber of states is finite. This system reacts to inputs from 
the environment in a probabilistic fashion: if J[. is in 
state s and receives a from the environment, the prob- 
ability that ^ transitions to state s' is Ps,a{s')- More- 
over, assume that the environment cannot observe the 
state of J{ in order to choose the particular input a. The 
analysis you want to perform on this system is to cal- 
culate a tight lower bound of the probability that the 
system achieves a certain goal, no matter what the in- 
puts are. For instance, inputs can model notifications of 
the (un)availability of resources, and you might want to 
check that your system sends a message with probabil- 
ity at least 0.8, no matter what the available resources 
are. 

The problem in the paragraph above can be modelled 
using Probabilistic Finite Automata (PFA) |[lll2l- The 
assumption that inputs do not depend on the internal 
state of the state of the input is central to assert that a 
PFA model adequately reflects the behaviour of the sys- 
tem. In case the environment can observe the state of 
J[ to choose the particular input a, the problem can be 
modelled using Markov Decision Processes (MDP) Js]]. 

The usual semantics for PFA rely on the concept of 
acceptance, by considering the set of finite words end- 
ing in an acceptance state with probability greater than 



a given cut-point rj. In contrast, we focus on the concept 
of reachabiUty, and we are interested on the probability 
with which each infinite word reaches some of the states 
in a given set S. In the realm of MDPs, both the supre- 
mum and the infimum probability can be calculated in 
polynomial time |4]. In contrast, in the PFA setting the 
supremum problem is undecidable |2] for both finite and 
infinite words Q. In fact, the supremum probability that 
^ reaches a state in .S cannot be even approximated al- 
gorithmically. This undecidability result was the key to 
prove undecidability results for MDPs under partial in- 
formation [5J as well as undecidability for Probabilistic 
Biichi Automata 

We present an algorithm to approximate the infimum 
probability that a PFA J{. reaches a set of states S. More- 
over, the computed value v is a lower bound of the in- 
fimum and, by performing a sufficient number of itera- 
tions, we can ensure that it is as close to the infimum as 
desired. Using the value v, we can answer our motivat- 
ing problem by stating that "the probability that the goal 
is achieved is at least v, no matter what the inputs are". 
The fact that the value v is close to the infimum implies 
that the bound we provide is tight. 



'Urquiza 1949 16/F. Rosario, (2000) Rosario, Argentina 



Here, we consider only infinite words, as the infimum probability 
over finite words is either 1 , if the initial state of the system is in 5, or 
0, if it is not. 
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2. Algorithm 

For our algorithm, we use the following definitions: a 
Probabilistic Finite Automata (PFA) is a quintuple J?l = 
(S, S, !P, s',S), where 5 is a finite set of states, E is a set 
of symbols, !P is a set of probability distributions on S , 
comprising one probability distribution Ps,a{-) for each 
pair (i, a)mS X E. The state s' is called the initial state 
of y[, and >S is a set of hitting states. We assume s' i S. 

A finite path in y[ is a sequence 

n - s'.ai.si.- ■ ■ .a„.s„ 

where a, € E and i, e S for all /. Note that paths al- 
ways start with the initial state s' . We write len(;7r) for n 
and last(7r) for s„. In an analogous way to finite paths, 
infinite paths are infinite sequences alternating symbols 
and states. The set of all infinite paths having the finite 
path n as prefix is denoted by n'^ . 

Given a word t// over E, let ij/[k.] denote the ^-th sym- 
bol in iff. For every infinite word t// over E, for every fi- 
nite path 71, the probability Pr'^(;7r^) is defined as 1 if = 
i';ifi/r[len(7r) + l] = a, we have Pr'^(7r.a'./) = Pr'^(7rT). 
Pia,st(;r),(i(^); if i/'[len(7r)+l] a, thenPr'*(;r.Q'.i'f) = 0. In 
the same way as for Markov chains and MDPs (namely, 
by resorting to the Caratheodory extension theorem), 
the previous definition for sets of the form can be 
extended in such a way that, for all infinite words if/, 
the value Pr '^{Z) is defined for all measurable sets Z of 
infinite paths. 

Let 'H be the set of all infinite paths p such that some 
of the states in p is in S. The amount we want to approx- 
imate is / = inf^f, Pt'^CH). Note that can be written 
as 

'H^\+jn\ (1) 

TieC 

where C is the set of all finite paths n: such that last(7r) 
is the only state of n in S. 

In order to approximate /, our algorithm iterates pro- 
ducing two values in each iteration r. One of the values 
is a lower bound Ir and the other one is an upper bound 
Ur. These bounds comply with: 
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To approximate / with error at most e, the algorithm 
stops when Ur-lr < £ (this is guaranteed to occur as both 



Ur and /,. converge to the same limit), and then returns Ir. 
Note that m,- is also a value with error less than e but, in 
order to give a safe lower bound on the probability that 
a hitting state is reached, we use the pessimistic value 

Ir < I. 

In the next subsections, we show how to calculate up- 
per and lower bounds complying with the desired prop- 
erties. 

2.1. Lower bounds 

Let 'Hr - l+Jc, where Cr is the set of paths such 
that last(7r) is the only state of tt in and len(7r) < r. By 
making the same observation as for Eq. [T] we deduce 
that 'Hr is the set of all infinite paths reaching S after at 
most r symbols. We often profit from the inclusion 

U £ U+i . 

We take Ir - inf^ Pr'^CT/^)- Next, we show that this 
number can be calculated by brute force. 

Since only the first r symbols are relevant, we need 
to consider each of the finite words w having exactly 
r symbols. The truncation operator tpir, that returns 
the prefix of i// having length r, will thus be quite use- 
ful in this subsection. In addition, we use the notation 
Pr ("T/r) to mean Pr '^{'Hr), where if/ is any infinite word 
such that ^[r - w. 

For each w with len(w) = r, we construct a fi- 
nite Markov chain M. The procedure resembles the 
standard unfolding of a probabilistic automaton (or an 
MDP) for a particular adversary [t?!], and so we merely 
outline it. The states of M are pairs {s,k) with s in 
S and < A: < r. To describe M briefly, let's say 
that the path s'.a\.s\. ■ ■ ■ .a„.s„ in J[. maps to the path 
{s',Q).{si,\).{s2,2).--- .{s„,n)mM. For all < k < r, 
the probability of transitioning from {s, k) to is' , k + \) 
is Ps.wik+^is') (note that these probabilities depend on 
w). For simplicity, the states is, r) are stuttering. The 
initial state of M is is' , 0). The previous definitions for 
M imply that the probabilities of the paths in J[ having 
length at most r coincide with the probabilities of the 
corresponding paths in M: 

Pr!^(s'.ai.ii. ■ ■ ■ .a„.Sn ) 

= Ps\a,{si) ■ Will Pst,atJSk+l) 

= PrMi(s\0).isul).(s2,2).--- .(s„,n)) . 

As a consequence, the probability that w reaches S in 
at most r steps equals the probability that Ai reaches a 
state in >S X {0, ■ ■ ■ , r). The latter probability can be cal- 
culated using standard techniques, as it poses a simple 
reachability problem for finite Markov chains. 
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We have just showed that is computable. We 
still need to prove that it complies with the proper- 
ties we need so that our main algorithm works. In or- 
der to prove Inequation |2] we use the fact that Ir - 
min„,j=5v. Pr "CHr), where Wr is the set of all words of 
length r. Let w* be arg min„(=vy. , Pr'^'CT/r+i) and w*"^ 
be w*ir. The required inequality l,- < Pr" ('7/r+i) fol- 
lows since 1^ = min^.^iv, Pr^'XH) < Pr"'* 'CK,) = 
Pr"'*('?Yr) < Pr^'CKr+i), where the last inequaUty holds 
since 'Hr £ 'J^r+i- 

Next, we prove Inequation [3] Let fi = 
((/'(m))J^^j be a sequence of infinite words such 
that Um,„^coPr'''*'"'('H,.) = / and the sequence 
{PT'^^'"\'Hr) is non-increasing (such a sequence ex- 
ists by definition of infimum). Let w* be a word of 
length r that appears infinitely often in the sequence 
((/'(m)Xr)J^^j (this word exists as the sequence is infi- 
nite, and there are finitely many words of length r). 

We prove Inequation |3] by proving Pr "' (TY^ ) < /. 
Suppose, towards a contradiction, that Pr"' ("Kr) > /. 
Then, by definition of yu there exists 4r(p) in fi such 
that Pr"'*('?Y,) > Vx'l'^P\'Hr) > I. Since w* appears 
infinitely often in ( (/'(m)4,r there exists q > p 
such that ij/(q)ir = w*. Since the values ifrim) are 
non-increasing, we reach the following contradiction: 
Pr"''CH,) > Pr'/'^P^'Hr) > Pr '*<'?'(•?/,) = Pr'^'CH,)- 

It remains to prove Eq.|4l In other to prove this equal- 
ity, let *!* be the sequence 

(>I', = argminPr"'(n))r-i 

weWr 

(the set Wr has been defined above). Note that 

Z, = Pr*'(U). (8) 

Given we construct an infinite limit worc|^ hav- 
ing the property that, for every M, the prefix lij/iM ap- 
pears infinitely often in the sequence (TriM)^!!^^. We 
take the first symbol to be any symbol that ap- 
pears infinitely often in ("PtJ,! )^j. In order to obtain 
the second symbol i/'[2], we consider the subsequence 
*P' of *!* containing all words in *!* whose first sym- 
bol is if [I]. Then, (/'[2] is any symbol that appears in- 
finitely often as the second symbol in ( ^ll2 )^2- 
general, we can describe the process to obtain *I"*^ and 
i^[M] in an inductive fashion, by stating that i/'[M] is any 
symbol that appears infinitely often in (*I'^"'[M] )^^ 
and is an (infinite) subsequence of ^P*^ ' comply- 
ing with ^^[M] = i/'[M]. The existence of the subse- 
quence *P*^ ensures that i//lM appears infinitely often in 
(^',lM)%^,as desired. 



^We use the word limit as it resembles the limit schedulers in 



As an auxiliary result, we prove Pr'^i'H) - I. Sup- 
pose, towards a contradiction, that Pr'^i'H) > I. Then, 
there exists i/r' such that PrH^H) > Pr'^'('H) > /. A^ 

ViA : Pr^CT/) = Hm Pr^CKt) , (9) 
there exists K such that 

PxH'Hk) > Pr^'CT/) > Pi'^'CHm) = Pr^'^^'CKM) 

(10) 

for all M. By definition of i/r, there exists M > K such 
that "VmIK = i/^iK. Then, Pr^CKjf) = Pt'I'^^CHk) = 
Pi.'i'«i^(7Y^) < Pr*«(<HM) < Pr^'-^'^CT^Af) (where the 
last inequality holds by definition of *Pm) thus contra- 
dicting InequationfTOl 

Now we are ready to prove lim^^co Ir = I- Since Ir < I 
for all r, we have lim^-^oo Ir ^ I- Suppose, towards a 
contradiction, that lim^^co h < I- Then, by Pr'^('H) = / 
and Eq.|9l there exists K such that 

lim Ir < Pr '^{^Hk) = Pr ^^^C^/r) ■ ( 1 1 ) 

By definition of i/', there exists M > K such that 
^mIK - il^iK. Then, by Eq. [8l we have limr^oo h ^ 
Pr*"CKM) > Pr'*'"-^^CK^) = Pr^^^CK^), which con- 
tradicts InequationfTTl 

2.2. Upper bounds 

For our upper bounds, we use lasso-shaped words 
(LSW). A LSW is an infinite word of the form iff - 
a\ ■ ■ ■ UKiPi ■ ■ -PMf, in which the last M in which the 
sequence of symbols - ■ ■ /3m is looped infinitely many 
times. The name lasso-shaped is borrowed from the 
counterexamples for LTL properties of Biichi automata, 
this name being used, for instance, in |9]. Such coun- 
terexamples also consist of a finite stem and a sequence 
that is looped infinitely many times. 

In this paper, we restrict to LSWs with M < 2'^' (re- 
call that S is the set of states of the PEA), and we say 
that K is the order of iff, denoted by order(i/r). Note that, 
because of our restriction on the length of the loop, the 
amount of LSWs with order at most K is finite. 

We denote by LSW(r) the set of all LSW with order 
at most r. The set of all infinite words is denoted by 
Words. 

For upper bounds, we take Ur - inf^eLSW(/) Pr'^i'H). 
Inequalities |5] and |6] follow from LSW(r) c 
LSW(r-H 1) c Words. 



^^This equality is standard for reachability properties, and can be 
deduced from Pr^CK) = Pr*(l+)^^i \ 'Hn) = I^'j^L^Fi'^CHt \ 
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The computability of Ur follows in a similar way to 
that of Ir'. the amount of LSWs having order at most r 
is finite, and we can explore the probabilities for each 
of these words. Similarly as for the lower bounds, the 
probability for a word w\{w2)" is calculated by con- 
structing a finite Markov chain. We just outline the con- 
struction. The set of the states of the Markov chain is 



5x11, 



, maMK, M] ] X {S, L) 



(where K - len(wi) and M - len(w2)). The initial state 
is (5',1,S). In the state {s,n,S) ((s, n, L), resp.), the 
probability distribution for the next state is determined 
by the «-th symbol in the stem (in the loop, resp.) In 
symbols, the probability of transitioning from (s, n, S) 
to is' ,n + 1,S) is Ps,wi[n](/) whenever n < K. From 
is, K, S) to is',K, L), the probability is pj,,r,[A:](i')- The 
probabilities for the loop are defined in a similar way: 
the only difference is that in a state {s, M, L) in the end 
of the loop, we have that Ps,w2[M]{^') is the probability of 
transitioning to {s, 1, L) (that is, we return to the begin- 
ning of the loop). Note that all the paths with positive 
probability are of the form 

isuhS)---{sK,K,S) 

■ ■ ■ (SK+iM+l, 1 , L), ■ ■ ■ , (SK+iM+M, M,L) ■■■ ■ 

Is is easy to see that the probability Pr'"i("'2>"(>K) is the 
probability of reaching a state (s, n, I) such that s € S, 
and so the minimum probability for all words of order at 
most K can be obtained by constructing a Markov chain 
for each of such words. 

It remains to prove Eq. [T] If / = 1, then = 1 for 
all r, and so the equation is trivial. From now on, we 
concentrate on the case I < 1 . In order to prove that the 
limit is the infimum, it suffices to show that, for all e, 
there exists R such that 



inf Pr'^'CK) </-He 

i/(eLSW(R) 

We can indeed restrict to e such that 
e< 1 -/. 



(12) 



(13) 



(Having proved the result for such values, the result also 
holds for the values e' such that e' > 1 - /, by tak- 
ing e such that e - (1 - I)/2 < I - I < e' and hence 
inf^eLSW(«) Pr H^^) < I + e < I + e' .) 

We prove Inequation [12] by showing that there exists 
I/,* = wxiwiY with len(w2) < 2^ such that ¥r'l''('H) < 
/ H- e. By taking R to be the order of we obtain In- 
equation[T2] that is, the value ur is e-close to /. 




Figure 1 : Avoiding S with higli probability 



Let i/r^^^ be an infinite word such that Pi:'^''^('H) < 
I + e/2 (such a word exists by definition of infimum). 
Using this word, we construct the word tfr* with the de- 
sired properties. For this construction, we focus on the 
probability of not reaching >S (that is, the probability of 
all infinite paths such that none of the states is in S). By 
definition of i/''^^^, we know that t//^^^ does not reach S 
with probability greater than 1 - e/2 - /; in symbols: 



Pr'*"'(^'H) > 1 - e/2-/ 



(14) 



where -iTY is the complement of "H, that is, the set of all 
infinite paths p such that p[k] i S for all k. 
Using i/''^^^, we define i//* in such a way that 



Pr'^*(^'7/) > 1 - e-/ 



(15) 



and so Pr CH) < I + £. The proof proceeds by finding 
numbers K and M such that the first K + M symbols 
of i/r* are the same as in tfr'^^^. We name these symbols 
ai,a2, ••■ ,o!k,/3\,/32,--- ,/3m- After these symbols, the 
word t//* repeats ■ ■ ■ ,/5m indefinitely. This word is 
illustrated in Fig. [1] The intuition behind the proof is 
that there exists a set Qi of states such that, after ex- 
actly K steps, there is sufficiently high probability to be 
in Qi, without hitting S (in the figure, states in S are 
represented with crosses). Moreover, if (Qi, re- 
spectively) is the set of all states that can be reached 
after symbol j8, (/3m, resp.) occurs in some state in Qi 
(Qm, resp.), then g, n >S = for all 1 < / < M. We find 
K, M and show that i//* complies with InequationfTSl 

In order to obtain the required K, M, we profit from 
the fact that a PFA according to our definition can be 
seen as a particular case of an MDP For the sake of 
completeness, we show how our definition for PFA 
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matches the definition of MDP in |10]. If the MDP 
underlying a PFA J[. is obvious to the reader, then the 
rest of this paragraph can be safely skipped. In ifioll . 
(Def. 3.1), an MDP n = {S,A,p) is defined by a set of 
states 5, a set of actions A{s) enabled at each state s, 
and probabilities PsAa) of stepping from i to f using a, 
for each a e A(s). When mapping a PFA J[ to an MDP 
n, the set of states S of 11 is the same set of states as in 
for each s the set A{s) of actions enabled is the set 
2; the probabilities Pst{a) in ifioll are simply Ps,a(t)- 

Using the MDP underlying yi, we can resort to the 
end-component theorem ( 1,10. Thm. 3.2]). In terms of 
PFA, the definition of an end component is as follows. 

Definition 1. An end component is a set £ c 5 x 2 such 
that for every states s\ + s„ in (a pair in) E there exists 
a path si.ai.S2- ■ ■ ■ .a„-\.Sn such that {sk,ak) e E and 
Psk,atisk+\) > for all 1 < k < n-1. We write States(£') 
for the set of states of E. When no confusion arises, we 
simply write s e E instead of i e States(£'). 

Let & be the set of infinite paths si.ai.S2- - ■ ■ such that 
there exists T such that the set {{s,, a,) | f > T} is an end 
component. The end-component theorem states that fi 
has probability 1 for all words. The paths in £ are said to 
end in an end component. Then, the set of paths that do 
not end in an end component (that is, the paths for which 
no such T exists) has probability for all words and, 
roughly speaking, can thus be disregarded in probability 
calculations. 

From now on, we are interested on the set S com- 
prising all paths ending in an end component. Now 
we show a partition for fi. For all finite paths n, end 
components E, let Z(7r, E) be the set of all infinite paths 
n.a\.S2.a2- ■ ■ ■ such that (ak, Sk+\) e E for all k. Next, 
we prove that the set & is equal to &' - 1+J(;r,£)ez ^) 
where Z, is the set of all pairs (tt, E) such that n is either 
the trivial path s', and s' e E; or n - s'. ■ ■ ■ .s„-i.a„.s„ 
and {Sn-i,a„) i E and s„ in E. In words, the last 
state/symbol pair is not in E, but the last state is. 
Clearly, the inclusion £' c £ holds as the paths in 
Z(jT,E) end in E for all n, E. In order to prove the 
inclusion £ c £', we prove that any path i/r e £ is 
also in £'. Since tfr e S, there exists T as in Def. [1] 
Let's consider the minimum such T. The existence 
of T ensures that p has a prefix n after which all the 
pairs state/symbol are in E. Moreover, since we are 
considering the minimum T, either n is the trivial path 
s', and s' is in E; or the last state/symbol pair before 
last(7r) is not in E. In summary, the fact that R is min- 
imum ensures that there exists in,E) e Z, such that 
p e Z(n, E). It remains to prove disjointness, that is. 




Figure 2: (s[ , a) is in E, but (.V2, a) is not 



Z(7T,E) nZ(7T',E') + ©imply {n,E) = (n' ,E'). Sup- 
pose that there exists p 6 Z(n,E) n Z{n' ,E'). The set 
of all state/symbol pairs that appear infinitely often in 
p are all the pairs in £ (as p € Z{ji, E)), and the same 
goes for £", thus yielding E = £". It remains to prove 
71 - n' . We have that n and n' are both a prefix of p. 
Moreover, since we consider only finite paths in which 
the last state/symbol pak is not in E, we have that n is 
the smallest prefix such that after n all the state/symbol 
pairs are in E, and the same holds for n' . Then, both n 
and n' have the same length, and son - n' . 

As a consequence of the partition we found, and 
the end-component theorem, for all words we have 
Pr^il) = Pr^fi) = Y.nY.mn.E)eZ\ ^^^{Z{7:,E)). If a 
paths ends in an end component E and does not hit S, 
then no prefix hits S, and E has no states in S. Hence, 
for all words ^ we have 

Pr^^'7/)= Yj Z Pi-^(Z(^,£))- 

{ji\nnS=<b\ {E\(n.E)£Zl\EnS=lt\ 

The outer sum ranges over all finite paths such that no 
state is in S (which we denote as tt n >S = 0), and the 
inner sum ranges over all end components E such that 
the last state/action pair in n is not in E, the last state is 
in E, and no state of £ is in >S (denoted by Er\S - 0). In 
particular, for the word ^"^1^ in Inequation [141 we have 
Pr'A"^(^'H) = E.n5=0i:(.,£)EZA£n5=0Pr^"'(Z(^,£)) > 
l-e/2-I. Then, there exists a finite setS Q {n \ n i S} 

such that 2;jn5=0 Yj(n.E)eZ/\EnS=(li 

|e - /. Let B = maX;,^^ len(7r). For the sake of brevity, 
let be the set of all pairs (n, E) such that n nS = (/>, 
and len(;r) < B, and (tt, E) e Z, and E n S ^ 0, and 
Pr^"'(Z(7r,£)) > 0. Then, 

2 Pr^"'(Z(7r,£))> 1-^e-/. (16) 

(7r,E)eV 

Note that we can restrict to the pairs (n, E) such that 
Pr '*'''( Z(7r, £) ) > 0, as the pairs with probability do 
not affect the sum. In addition, by Inequation [13] we 
have 1 - e - / > 0, and so in the sum in Inequation [16] 
there is at least one positive summand Pr ( Z(7r, E) ). 

The desired K, M are now obtained from i//'^^^ and 
B. Note that, although we restricted to the summands 
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complying with Pr ( Z(n, E)) > 0, it is still possible 
that exits E after n with positive probability (as the 
same symbol might be inside E for a reachable state 
s, but outside E for a state s' that is reachable after the 
same number of steps as s, see Fig.|2]l. We show that, by 
considering arbitrarily large paths, the probability that E 
is exited becomes arbitrarily small. 

Let b'^is) be the probability that, after k steps, the 
state is reached is s. We generalize this notation to sets 
of states. Formally: fe^(r) = i;ien(;r)=/t,iast(;r)e7- Pr"'(7r^)- 
We call the distribution b'^(-) a belief state, follow- 
ing the nomenclature for POMDPs [1 1.1. Since the 
set of states is finite, there exist two indices x < y 
such that supp(/7^ ) - supp(/7f ) (where supp denotes 
the support of the distribution). Moreover, given any 
two numbers X, Y, such that Y > X + 2'"^', we have 
X < X < y < Y and supp(/7^ ) = supp(Z7y ) for 
some X, y. Since the amount of sequences of the form 
Tqji ■ ■ -yvTv with V < 2'"^' is finite (where each T,, is 
a set of states), at least one of such finite sequences 
appears infinitely many times in the infinite sequence 
supp(bf')i^''\l) supp{bf'w'\2) Suppose this 
finite sequence is cr = Tqji ■ ■ ■ jvTv- 

We show that we can take ■ ■ ■ ,13m - Jx, - ■ ■ ,yv- 
In addition, we take to be a number (defined below) 
greater than B, in which an occurrence of cr starts. 

Given a component £ in a pair in 'V (defined before 
IneguationfTSIl. let 

e To n States(£') | Vv < V : 
Pvy'-^'is'.yi.si.--- .y,..s,)>0 ^ e E} . 

In other words, comprises the states in roriStates(£') 
from which, when executing ji ■ ■ ■ jy, we can only 
reach states in E. Let Q"^ be States(£') \ . 

Consider the infinite sequence e(l), e(2) ■ of indices 
where cr starts. We show that lim^^co b'^, As) - for all 
s € > 0. (As the number of states is finite, this 
implies lim,,^oo ^f(,,)(2^^) = 0-) Suppose, towards a 
contradiction, that for some s e 2^^, Z > 0, we have 
fe^(,,)(.?) > / for all V. By definition of QT^ , there exists 
s i E, d > Q, and v such that Pr''' " ''' (i.yi ■ ■ ■ y^-s') - d. 
Then, the probability of staying in E after the n-th 
repetition of cr is less than or equal to (1 - (/ ■ d))", 
for all n. This implies that the probability of staying 
in E indefinitely is 0, thus contradicting the fact that 
Pr'^"'(Z(7r,£))>0. 

As a result, for all pairs {n,E) in 'V, there exists 
e{7i, E) e {e(l), e(2), ■ ■ ■ ) such that 

b%]E)(Q^^) < ■ 1^1) ■ (17) 



Approximate/Compute 


Infimum 


Supremum [2] 


Reachability 


V/? 


x/x 


Acceptance [12] 


?/x 


x/x 



Table 1 : Existence of algorithms for PFA 



Define K = max(B, {e(;r, £) | (7:,E) e O')) and 
Y(7T,E) = Z(n,E) \ {p I p[K] e Q^^). We have 
Z(7r, E) c Yin, E) U {tt^ | n[K] e Q^'^ A len(7r) = K}. 
Then, 

Pr'*(Z(7r,£)) < PT'^(Y{7T,E)) + bl(Q-^'^) (18) 

for all infinite words t//. 
We have 

Pr'**(F(7r,£)) > Pr-^''^ y(7r, £) ) (19) 

as, under i/r* , all paths of length K ending in a state in 
do not reach states outside E (because of our definition 
of and the symbols y,,). In fact, if len(7r) > K, the 
scenario in Fig.|2]is possible under iff'^^^, but not possible 
under tfr*. Roughly speaking, after K steps the word 
i/r* does not escape E, thus yielding higher (or equal) 
probability for Y(n, E) than any word i// such that ifriK = 
tfr*iK and, in particular, than i//'^^^. Then, 

> { Y{7T, E) c Z{7T, E), the sets Z(7t, E) partition -^'H} 
Zi.,E)e'vPr'^'(Y(n,E)) 

> {Ineq.[l9)2:(,,£),^Pr'A"^(y(7r,£)) 

> {Ineq.im Z(..E)e'v'Pr'^'\Z(^,E)) - bfiQ-") 

> {Inequations[T6][l2) 1 - |e - / - ^ ■ £/(4-\^\) 

> l-e-I 

In conclusion, the word i/r* = ai ■ ■ ■ UKifii ■ ■ ■ Pm)'^ 
(where ai ■ ■ ■ ukPi ■ ■ -Pm are first K + M symbols in 
i//^^^) complies with Inequation [15] Since order(i^*) - 
K, we obtain inf^eLSW(A') Pr'''('H) < I + e. By Inequa- 
tion|5j this inequality implies inf^!,gLSW(yt) Pi'^i'H) < I+e 
for all k > K, thus ensuring Eq.|2] 

3. Discussion 

Our algorithm is nonprimitive recursive, and we have 
still nothing to say about the complexity of the problem. 

However, the fact that there exists an algorithm to ap- 
proximate the value is quite surprising considering sim- 
ilar problems for PFA, as shown in Table [1] The table 
indicates, for the problems of reachability and accep- 
tance, whether there exists an algorithm to approximate 
and/or to compute extremal values. Note that the only 
^J in the table corresponds to the result in this paper 
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The table also indicates two pending questions: whether 
there exists an algorithm to effectively compute the in- 
fimum for reachability, and whether the infimum for ac- 
ceptance can be approximated. 

The undecidability for the supremum probability has 
been used to prove that quantitative model checking un- 
der partial information ||5l|l3|] is undecidable for proper- 
ties involving the supremum. The setting of these papers 
is more general, as several entities might have different 
information about the state of the system (in contrast, 
the problem we address in this paper concerns only an 
environment that has no information about the state of 
the system). However, we expect that the proof we pre- 
sented sheds some light on whether this more general 
problem is computable or not, in case we consider the 
infimum instead of the supremum. 



[13] S. Giro, Undecidability results for distributed probabilistic sys- 
tems, in: M. V. M. Oliveira, J. Woodcock (Eds.), SBMF, Vol. 
5902 of Lecture Notes in Computer Science, Springer, 2009, 
pp. 220-235. 



References 

[1] M. O. Rabin, Probabilistic automata. Information and Control 
6 (3) (1963) 230-245. 

[2] O. Madani, S. Hanks, A. Condon, On the undecidability of prob- 
abilistic planning and related stochastic optimization problems, 
Artif. Intell. 147 (1-2) (2003) 5-34. 

[3] M. L. Puterman, Markov Decision Processes: Discrete Stochas- 
tic Dynamic Programming, John Wiley, 1994. 

[4] A. Bianco, L. de Alfaro, Model checking of probabalistic and 
nondeterministic systems, in: P. S. Thiagarajan (Ed.), FSTTCS, 
Vol. 1026 of Lecture Notes in Computer Science, Springer, 
1995, pp. 499-513. 

[5] S. Giro, P. R. D'Argenio, Quantitative model checking revis- 
ited: Neither decidable nor approximable, in: J.-F. Raskin, P. S. 
Thiagarajan (Eds.), FORMATS, Vol. 4763 of Lecture Notes in 
Computer Science, Springer, 2007, pp. 179-194. 

[6] C. Baier, N. Bertrand, M. GroBer, On decision problems 
for probabilistic Biichi automata, in: R. M. Amadio (Ed.), 
FoSSaCS, Vol. 4962 of Lecture Notes in Computer Science, 
Springer, 2008, pp. 287-301. 

[7] R. Segala, Modeling and verification of random- 
ized distributed real-time systems, Ph.D. thesis, Lab- 
oratory for Computer Science. MIT, available at 
[profs . sci .univr . it/-segala/www/pub/phd. tar .gz, 
(1995). 

[8] S. Giro, P. R. D'Argenio, On the expressive power of schedulers 
in distributed probabilistic systems, Electr Notes Theor Com- 
put. Sci. 253 (3) (2009) 45-71. 

[9] V. Schuppan, A. Biere, Shortest counterexamples for symbolic 
model checking of Itl with past, in: N. Halbwachs, L. D. Zuck 
(Eds.), TACAS, Vol. 3440 of Lecture Notes in Computer Sci- 
ence, Springer, 2005, pp. 493-509. 
[10] L. de Alfaro, Formal verification of probabilistic sys- 
tems, Ph.D. thesis, Stanford University, available at 
users . soe .ucsc . eduZ-luca/papers/thesis/thesis .pdJ 
(1997). 

[11] L. P. KaelbUng, M. L. Liftman, A. R. Cassandra, Planning and 
acting in partially observable stochastic domains, Artif. Intell. 
101 (1-2) (1998) 99-134. 

[12] V. D. Blondel, V. Canterini, Undecidable problems for prob- 
abilistic automata of fixed dimension, Theory Comput. Syst. 
36 (3) (2003) 231-245. 



7 



